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Highlights 


Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD) 
provides  medical  care  to  9.6  million 
active  duty  service  members,  their 
families,  and  other  eligible 
beneficiaries  worldwide.  DOD’s 
Military  Health  System  has  long  been 
engaged  in  efforts  to  acquire  and 
deploy  an  electronic  health  record 
system.  The  latest  version  of  this 
initiative — the  Armed  Forces  Health 
Longitudinal  Technology  Application 
(AHLTA) — was  expected  to  give 
health  care  providers  real-time  access 
to  individual  and  military  population 
health  information  and  facilitate 
clinical  support.  However,  the 
system’s  early  performance  was 
problematic,  and  DOD  recently  stated 
that  it  intended  to  acquire  a  new 
electronic  health  record  system.  GAO 
was  asked  to  (1)  determine  the  status 
of  AHLTA,  (2)  determine  DOD’s  plans 
for  acquiring  its  new  system,  and  (3) 
evaluate  DOD’s  acquisition 
management  of  the  initiative.  To  do 
this,  GAO  reviewed  program  plans, 
reports,  and  other  documentation 
and  interviewed  DOD  officials. 


What  GAO  Recommends 

GAO  is  recommending  that  DOD  take 
six  actions  to  help  ensure  that  it  has 
disciplined  and  effective  processes  in 
place  to  manage  the  acquisition  of 
further  electronic  health  record 
system  capabilities.  In  written 
comments  on  a  draft  of  this  report, 
DOD  concurred  with  GAO’s 
recommendations  and  described 
actions  planned  to  address  them. 


View  GAO-1 1-50  or  key  components. 
For  more  information,  contact  Valerie  C. 
Melvin  at  (202)  512-6304  or 
melvinv@gao.gov. 


What  GAO  Found 

After  obligating  approximately  $2  billion  over  the  13-year  life  of  its  initiative  to 
acquire  an  electronic  health  record  system,  as  of  September  2010,  DOD  had 
delivered  various  capabilities  for  outpatient  care  and  dental  care 
documentation.  DOD  had  scaled  back  other  capabilities  it  had  originally 
planned  to  deliver,  such  as  replacement  of  legacy  systems  and  inpatient  care 
management.  In  addition,  users  continued  to  experience  significant  problems 
with  the  performance  (speed,  usability,  and  availability)  of  the  portions  of  the 
system  that  have  been  deployed.  DOD  has  initiated  efforts  to  improve  system 
performance  and  enhance  functionality  and  plans  to  continue  its  efforts  to 
stabilize  the  AHLTA  system  through  2015,  as  a  “bridge”  to  the  new  electronic 
health  record  system  it  intends  to  acquire. 

According  to  DOD,  the  planned  new  electronic  health  record  system — known 
as  the  EHR  Way  Ahead — is  to  be  a  comprehensive,  real-time  health  record  for 
service  members  and  their  families  and  beneficiaries.  The  system  is  expected 
to  address  performance  problems,  provide  unaddressed  capabilities  such  as 
comprehensive  medical  documentation,  capture  and  share  medical  data 
electronically  within  DOD,  and  improve  existing  information  sharing  with  the 
Department  of  Veterans  Affairs.  As  of  September  2010,  the  department  had 
established  a  planning  office,  and  this  office  had  begun  an  analysis  of 
alternatives  for  meeting  the  new  system  requirements.  Completion  of  this 
analysis  is  currently  scheduled  for  December  2010.  Following  its  completion, 
DOD  expects  to  select  a  technical  solution  for  the  system  and  release  a 
delivery  schedule.  DOD’s  fiscal  year  2011  budget  request  included  $302  million 
for  the  EHR  Way  Ahead  initiative. 

Weaknesses  in  key  acquisition  management  and  planning  processes 
contributed  to  AHLTA  having  fewer  capabilities  than  originally  expected, 
experiencing  persistent  performance  problems,  and  not  fully  meeting  the 
needs  of  users. 

•  A  comprehensive  project  management  plan  was  not  established  to  guide 
the  department’s  execution  of  the  system  acquisition. 

•  A  tailored  systems  engineering  plan  did  not  exist  to  guide  the  technical 
development  of  the  system,  an  effort  that  was  characterized  by  significant 
complexity. 

•  Requirements  were  incomplete  and  did  not  sufficiently  reflect  user  and 
operational  needs. 

•  An  effective  plan  was  not  used  to  improve  users’  satisfaction  with  the 
system. 

DOD  has  initiated  efforts  to  bring  its  processes  into  alignment  with  industry 
best  practices.  However,  it  has  not  carried  out  a  planned  independent 
evaluation  to  ensure  it  has  made  these  improvements.  Until  it  ensures  that 
these  weaknesses  are  addressed,  DOD  risks  undermining  the  success  of 
further  efforts  to  acquire  electronic  health  record  system  capabilities. 
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GAO 

^^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


October  6,  2010 

The  Honorable  Judd  Gregg 
Ranking  Member 
Committee  on  the  Budget 
United  States  Senate 

Dear  Senator  Gregg: 

This  report  responds  to  your  request  that  we  examine  the  Department  of 
Defense’s  (DOD)  efforts  to  implement  its  military  electronic  health  record 
system  known  as  the  Armed  Forces  Health  Longitudinal  Technology 
Application  (AHLTA).  When  fully  deployed,  AHLTA  was  envisioned  to 
provide  the  department  with  a  modernized  health  information  system  that 
would  generate  and  maintain  a  comprehensive,  lifelong,  computer-based 
patient  record  for  every  soldier,  sailor,  airman,  and  marine;  their  family 
members;  and  others  entitled  to  DOD  military  health  care.  The  electronic 
health  record  was  expected  to  give  health  care  providers  real-time  access 
to  individual  and  military  population  health  care  information,  thus 
facilitating  clinical  decision  support  and  rationale  for  care  rendered  to  U.S. 
service  members  worldwide.  However,  after  more  than  a  decade  of  effort 
to  deliver  this  system,  the  department  has  recently  begun  planning  for  a 
new  electronic  health  record  system. 

At  your  request,  we  conducted  a  study  of  DOD’s  efforts  to  acquire  and 
implement  its  electronic  health  record  system.  Specifically,  our  objectives 
were  to  (1)  determine  DOD’s  status  in  implementing  AHLTA,  (2) 
determine  the  department’s  plans  for  acquiring  a  new  system,  and  (3) 
evaluate  the  department’s  acquisition  management  for  its  electronic  health 
record  system. 

To  accomplish  the  objectives,  we  reviewed  relevant  program 
documentation  and  interviewed  appropriate  DOD  officials.  Specifically,  to 
determine  the  status  of  the  AHLTA  project,  we  reviewed  project  plans  and 
status  reports.  To  determine  the  department’s  plans  for  acquiring  a  new 
electronic  health  record  system,  we  reviewed  relevant  planning 
documents,  including  an  initial  capabilities  document.  To  evaluate  the 
department’s  management  of  its  electronic  health  record  acquisition,  we 
compared  the  department’s  activities  for  project  management  planning, 
systems  engineering  management,  requirements  development  and 
management,  user  satisfaction  feedback,  and  acquisition  management 
with  DOD  guidelines  and  industry  best  practices. 
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We  conducted  this  performance  audit  from  September  2009  to  October 
2010  at  DOD  offices  in  Falls  Church,  Virginia,  and  Bethesda,  Maryland,  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  A  more  complete  description  of  our 
objectives,  scope,  and  methodology  is  provided  in  appendix  I. 


Background 


DOD  operates  a  worldwide  health  care  program,  through  which  it  provides 
medical  care  and  assistance  to  9.6  million  active  duty  service  members, 
their  families,  and  other  eligible  beneficiaries.  Its  health  care  operations 
are  significant,  involving  approximately  135,000  personnel  in 
approximately  700  Army,  Navy,  and  Air  Force  medical  facilities  in  12 
domestic  regions,  as  well  as  European,  Pacific,  and  Latin  American 
regions.  The  department’s  fiscal  year  2010  budget  for  providing  health  care 
services  was  about  $49  billion. 

DOD’s  health  care  program  is  a  responsibility  of  the  Office  of  the 
Undersecretary  of  Defense  for  Personnel  and  Readiness.  Within  the  Office 
of  the  Undersecretary  is  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Health  Affairs,  which  is  responsible  for  the  department’s  Military 
Health  System  (MHS)  program. 

MHS  has  two  missions:  wartime  readiness  (maintaining  the  health  of 
service  members  and  treating  wartime  casualties)  and  peacetime  care 
(providing  for  the  health  care  needs  of  the  families  of  active-duty 
members,  retirees  and  their  families,  and  survivors).  The  Assistant 
Secretary  of  Defense  for  Health  Affairs  establishes  policy  regarding  health 
care  for  all  DOD  beneficiaries  and  also  plans  and  budgets  for  health  care 
operations  and  maintenance.  At  the  same  time,  each  military  service  has 
its  own  medical  department  that  operates  medical  facilities  (referred  to  as 
military  treatment  facilities)  and  recruits  and  funds  military  medical 
personnel.  Currently,  the  military  treatment  facilities  include  59  military 
hospitals  and  650  medical  and  dental  clinics.  DOD  provides  about  half  of 
MHS  services  through  these  military  facilities,  supplementing  this  by 
contracting  for  health  services  with  civilian  contract  providers.  Active- 
duty  members  are  required  to  obtain  care  at  military  treatment  facilities  if 
such  care  is  available;  in  contrast,  retirees  and  dependents  may  obtain 
care  at  either  military  facilities  or  through  civilian  contract  providers. 
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History  of  DOD’s 
Electronic  Health  Record 
System 


To  facilitate  the  delivery  of  medical  services,  in  1988,  DOD  initiated  the 
acquisition  of  an  electronic  health  record  system  to  support  all  of  its 
hospitals  and  clinics.  This  system,  the  Composite  Health  Care  System 
(CHCS),  was  intended  to  be  the  primary  medical  information  system 
deployed  worldwide  to  support  the  department’s  hospitals  and  clinics. 
DOD  envisioned  that  it  would  provide  automated  support  for  patient 
administrative  functions  (such  as  registrations,  admission,  and 
disposition);  ordering  and  retrieving  results  of  laboratory  and  radiology 
procedures;  ordering  and  recording  prescriptions;  and  patient 
appointment  scheduling. 

CHCS  was  deployed  in  1993;  however,  it  was  supported  by  numerous 
stand-alone  medical  information  systems,  such  as  the  department’s 
Ambulatory  Data  System,  Preventive  Health  Care  Application,  and 
Nutrition  Management  Information  System, 1  and  was  not  designed  to 
facilitate  the  exchange  of  information  from  one  system  or  military 
treatment  facility  to  the  next.  Specifically,  CHCS  was  facility-centric,  in 
which  each  facility  stored  only  its  own  medical  information  for  patients 
using  different  data  standards.  Therefore,  if  a  medical  provider  wanted  to 
obtain  complete  information  about  a  patient,  a  query  would  have  to  be 
made  to  each  of  the  CHCS  locations — a  time-  and  resource-intensive 
activity.  Additionally,  when  a  patient  moved  to  another  region,  the 
electronic  records  did  not  transfer  across  the  CHCS  locations  because  of 
the  different  data  standards  at  each  location.  The  lack  of  an  integrated 
system  perpetuated  the  reliance  on  paper-based  records,  leading  DOD  to 
pursue  a  comprehensive  electronic  health  care  record. 

To  this  end,  in  1997,  the  department  initiated  the  CHCS  II  program  to 
address  the  need  for  a  comprehensive,  lifelong,  computer-based  health 
care  record  for  every  service  member  and  their  beneficiaries.  The  vision 
for  CHCS  II  was  to  provide  access  to  a  patient’s  health  care  information 
with  a  single  query  by  providers  in  military  treatment  facilities. 
Specifically,  with  this  system,  DOD  planned  to  provide  worldwide  access 
to  outpatient,  inpatient,  dental,  and  vision  records,  and  to  make  them 
available  24  hours  a  day,  7  days  a  week.  This  new  system  was  to  be 
accomplished  with  the  use  of  a  centralized  repository  of  all  health  care 


'Each  of  these  systems  provided  certain  patient-related  information.  For  example,  the 
Ambulatory  Data  System  captured  certain  outpatient  information  relating  to  diagnosis  and 
treatment;  the  Preventive  Health  Care  Application  contained  information  on  preventive 
health  services;  and  the  Nutrition  Management  Information  System  supported  therapeutic 
nutrition  therapy  and  medical  food  management. 
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information  derived  using  common  data  standards.  The  system  was  to 
build  on  capabilities  of  existing  systems,  subsuming  their  functionality 
over  time,  while  adding  new  functionality  to  meet  mission  needs. 

CHCS  II’s  architecture  was  to  be  an  open  system,  client-server  design  of 
three  levels:  the  user  (client)  workstation  at  various  DOD  locations,  the 
DOD  computers’  (servers’)  operating  system  and  storage  hardware  and 
software,  and  a  clinical  data  repository  at  a  remote  computing  center 
where  the  information  would  be  stored.2  The  department  had  planned  to 
connect  all  workstations  at  an  installation’s  hospital  or  clinic  to  the 
servers  through  the  installation’s  local  or  wide  area  network.  It  had 
planned  to  divide  the  system  acquisition  into  seven  software  releases  to  be 
delivered  incrementally  by  June  2006  at  an  estimated  cost  of  $4.3  billion 
(in  1998  dollars). 

The  department’s  original  plan  had  called  for  deploying  a  prototype 
system  in  October  1998  and  beginning  deployment  of  the  initial  version  in 
about  April  1999.  However,  the  department  did  not  meet  its  schedule  to 
deliver  initial  CHCS  II  system  capabilities  and  associated  mission  benefits 
by  April  1999;  it  reported  that  the  initial  deployment  was  delayed  by  6 
months  because  of  a  failure  to  meet  initial  performance  requirements  and 
changes  in  system  requirements. 

In  July  2000,  the  department  redefined  its  plans  for  the  system  to  include 
adopting  a  new  technical  architecture,  establishing  a  means  for  controlling 
changes  to  requirements,  and  committing  to  the  incremental  release  of 
system  capabilities.  It  also  delayed  the  decision  date  for  deploying  the 
initial  system  capabilities  (for  outpatient  documentation)  to  January 
2001 — 21  months  later  than  its  original  commitment  for  the  system. 

However,  the  department  did  not  meet  this  commitment,  and  subsequently 
established  a  new  plan  that  called  for  incrementally  deploying 
functionality  to  achieve  the  system’s  full  operational  capability.  Delivery  of 
the  system  was  to  commence  in  July  2003  and  was  to  be  completed  by 
September  2007,  yielding  four  blocks  of  capabilities  that  would 
incrementally  populate  the  system’s  electronic  health  record  at  a  revised 
estimated  life-cycle  cost  of  $3.8  billion  through  2017. 


2Open  systems  conform  to  industry  standards  so  that  commercial  products  can  easily  be 
used  and  support  costs  can  be  minimized.  A  client  is  usually  a  desktop  computing  device  or 
program  that  is  “served”  by  one  or  more  networked  computing  devices. 
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•  Block  1  was  to  make  outpatient  information  available  worldwide  on  a 
continuous  basis  through  the  electronic  health  record  system  (as  opposed 
to  CHCS  legacy  functionality  which  only  made  records  available  at  a  single 
location),  provide  encounter  documentation,  aid  in  order  entry/results 
retrieval,  assist  in  encounter  coding  support,  provide  alerts  and  reminders 
(such  as  drug  interaction  alerts  and  special  duty  status),  facilitate  role- 
based  security,  and  establish  a  health  data  dictionary  and  a  master  patient 
index. 

•  Block  2  was  to  provide  automated  clinical  practice  guidelines,  optometric 
documentation,  and  dental  documentation. 

•  Block  3  was  to  replace  CHCS  ancillary  functionality  for  results  retrieval 
and  order  entry  for  outpatient  encounters  such  as  laboratory  and 
automatic  pathology,  pharmacy,  and  radiology. 

•  Block  4  was  to  provide  for  inpatient  order  entry  and  management, 
including  inpatient  clinical  and  critical  care  documentation. 

When  delivered,  the  system  was  to  allow  users  to  create  and  store 
computer-based  patient  records  using  workstation-  and  computer-based 
software  packages.  Each  facility’s  workstations  and  servers  were  to  be 
connected  via  each  installation’s  local  or  wide  area  networks.  Further, 
each  installation  was  to  be  connected  through  a  wide  area  network  to  a 
defense  computing  center  where  the  patient  records  would  be  stored  in  a 
database  known  as  the  clinical  data  repository.  DOD  intended  that  medical 
providers  would  ultimately  be  able  to  access  a  patient’s  computer-based 
record  from  any  military  treatment  facility,  no  matter  where  the  patient 
was  being  or  had  been  treated. 

According  to  program  documentation,  the  department  began  worldwide 
deployment  of  Block  1  in  January  2004.  It  completed  the  deployment  of 
this  block  in  December  2006.  However,  program  officials  stated  that  users 
experienced  numerous  performance  problems  with  the  capabilities  that 
were  delivered,  which  impacted  its  usability,  speed,  and  availability. 
Specifically,  the  department  reported  experiencing  the  following  problems 
with  the  delivery  of  Block  1: 

•  Usability.  The  system  did  not  support  varied  clinical  workflow  to  meet 
the  needs  of  various  types  of  practitioners,  had  missing  or  incomplete 
clinical  capabilities  (e.g.,  consult  and  referrals  management,  ancillaries, 
specialty  workflow  support),  did  not  support  fully  unified  or  user- 
customizable  patient  data,  and  did  not  have  a  user-friendly  interface. 
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•  Speed.  The  system  did  not  have  the  speed  or  performance  to  efficiently 
support  the  clinicians’  workflow  in  certain  environments  and  was  affected 
by  problems  such  as  coding  and  infrastructure  which  impacted  its  speed. 

•  Availability.  The  system  was  not  reliable  on  a  24-hour-a-day,  7-day-a- 
week  basis;  it  had  no  backup  for  disaster  recovery;  and  the  data  repository 
experienced  system  shutdowns  and  functional  interruptions. 


As  a  result  of  the  system  problems  associated  with  Block  1,  DOD  set  a  new 
date  for  system  completion — September  2011 — and  increased  the 
projected  life-cycle  cost  of  the  system  to  approximately  $5  billion,  which  it 
attributed  primarily  to  the  need  for  increased  operations  and  maintenance 
for  Block  1. 

The  department  also  took  a  number  of  other  steps  with  regard  to  the 
initiative.  Specifically,  in  May  2005,  it  terminated  plans  for  deploying  the 
Block  4  inpatient  functionality  with  the  intent  of  moving  this  functionality 
into  Block  3.3  However,  due  to  continuing  performance  problems  with  the 
functionality  that  had  been  delivered,  and  because  the  Block  3  deployment 
had  exceeded  the  department’s  5-year  limit  for  achieving  initial 
operational  capability  by  January  2008,  DOD  terminated  Block  3 
(laboratory,  radiology,  and  pharmacy)  as  well.  This  action  left  only  one  of 
the  four  planned  blocks — Block  2 — for  implementation.  Although  the 
department  reduced  the  scope  of  the  initiative  to  only  two  blocks,  the 
estimated  life-cycle  costs  were  revised  back  to  the  original  $3.8  billion 
(through  2021).  However,  the  department  encountered  performance 
problems  with  the  Block  2  dental  module  as  well  and,  in  December  2009, 
MHS  senior  leadership  implemented  a  strategic  pause  in  its  further 
deployment. 

Beyond  these  actions,  the  department  took  other  steps  over  the  course  of 
the  initiative.  Specifically,  in  November  2005,  the  Assistant  Secretary  of 
Defense  for  Health  Affairs  announced  a  change  in  the  name  of  the  system 
from  CHCS  II  to  AHLTA,  but  did  not  give  a  specific  reason  for  doing  so. 
Further,  as  part  of  its  attempt  to  improve  the  system,  DOD  awarded 
several  contracts  between  fiscal  year  2006  and  fiscal  year  2009  for  a  total 


3Military  hospitals  currently  use  Essentris,  a  commercial-off-the-shelf  product,  to  document 
inpatient  encounters  that  were  originally  planned  for  Block  4.  As  of  March  2010,  inpatient 
functionality  was  deployed  at  29  sites,  representing  62  percent  of  the  Military  Health 
System’s  inpatient  beds. 
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of  approximately  $40  million  to  address  performance  problems  and 
implement  software  enhancements.  The  contractors  began  deployment  of 
these  software  enhancements  (which  DOD  referred  to  as  AHLTA  3.3)  in 
December  2008. 


DOD’s  Acquisition  Process 
for  Its  Electronic  Health 
Record 


To  acquire  its  electronic  health  record  system,  DOD  used  several 
contractors  and  types  of  contracts.4  These  included  fixed-price,  time-and- 
materials,  and  cost-plus-fixed-fee  contracts,  each  of  which  involved  a 
different  level  of  cost  or  performance  risk  for  the  government.  5  The  prime 
developer  and  lead  integrator  for  CHCS  II,  Integic  (acquired  by  Northrop 
Grumman  in  2005),  was  awarded  a  time-and-materials  contract  for  about 
$65.4  million  in  1997  and  was  tasked  to  perform  systems  engineering, 
requirements  analysis,  architecture  evaluation,  software  design  and 
development,  engineering  and  development  testing,  test  and  evaluation, 
maintenance,  site  installation  and  implementation,  and  training.  Contracts 
for  system  development  and  integration  continued  through  fiscal  year 
2009. 


DOD  also  used  noncompetitive  contracts6  for  the  development  of  the 
system.  According  to  the  program  office,  11  noncompetitive  contracts  and 
task  or  delivery  orders,  totaling  approximately  $44.6  million,7  were 
awarded  for  the  system  from  fiscal  year  2004  through  fiscal  year  2012. 
Program  officials  stated  that  the  noncompetitive  contracts  were  awarded 
on  the  basis  that  (1)  DOD’s  need  for  the  supplies  or  service  was  so  urgent 
that  providing  each  awardee  under  a  multiple  award  contact  a  fair 


4We  have  identified  DOD  contracting  in  our  high-risk  list  since  1992,  and  DOD  business 
systems  modernization  as  high  risk  since  1995;  however,  we  did  not  explicitly  identify 
DOD’s  health  care  information  technology  procurement  processes  as  high-risk  areas.  See 
GAO,  High-Risk  Senes:  An  Update,  GAO-09-271  (Washington,  D.C.:  Jan.  22,  2009). 

5A  fixed-price  contract  provides  for  a  firm  price  or,  in  appropriate  cases,  a  ceiling  or 
adjustable  price.  A  time-and-materials  contract  provides  for  acquiring  supplies  or  services 
on  the  basis  of  direct  labor  hours  at  specified  fixed  hourly  rates  that  include  wages, 
overhead,  general  and  administrative  expenses,  and  profit  and  actual  cost  of  materials.  A 
cost-reimbursement  contract  provides  for  payment  of  allowable  incurred  costs,  to  the 
extent  prescribed  in  the  contract. 

6The  Federal  Acquisition  Regulation  allows  for  contracts  awarded  without  full  and  open 
competition  under  certain  circumstances  and  requires  written  justification  that  addresses 
these  circumstances. 

The  noncompetitive  contracts’  costs  are  about  2  percent  of  obligations  of  approximately 
$2  billion. 
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opportunity  would  have  resulted  in  unacceptable  delays;  (2)  only  one 
awardee  was  capable  of  providing  the  supplies  or  services  required  at  the 
level  of  quality  required  because  the  supplies  or  services  ordered  were 
unique  or  highly  specialized;  or  (3)  an  order  was  a  logical  follow-on  to  an 
order  already  issued  under  the  contract.8 

According  to  AHLTA  program  documentation,  the  system  acquisition  was 
guided  by  the  defense  acquisition  system,  which  is  documented  in  the 
department’s  DOD  5000.02  Instructions.  The  defense  acquisition  system 
consists  of  five  key  program  life-cycle  phases  and  three  related  milestone 
decision  points  that  major  acquisitions  must  meet  in  order  to  proceed  to 
the  next  phase  of  the  acquisition.9  At  each  milestone  point,  the  program  is 
reviewed  by  a  milestone  decision  authority  to  determine  whether  it  can 
move  to  the  next  life-cycle  phase. 

The  five  phases  of  the  defense  acquisition  are  as  follows: 

1.  Materiel  solution  analysis:  The  purpose  of  this  phase  is  to  assess, 
through  an  analysis  of  alternatives,  potential  solutions  to  satisfy  an 
approved  capability  need. 

2.  Technology  development:  The  purpose  of  this  phase  is  to  determine 
and  mature  the  appropriate  set  of  technologies  to  be  integrated  into 
the  investment  solution  by  iteratively  assessing  the  viability  of  the 
various  technologies  while  simultaneously  refining  user  requirements. 
To  enter  this  phase,  a  program  must  have  an  approved  analysis  of 
alternatives  and  pass  milestone  A.  To  exit  this  phase,  the  acquisition 
must  demonstrate  affordable  technology. 

3.  Engineering  and  manufacturing  development:  The  purpose  of  this 
phase  is  to  develop  a  system  or  an  increment  of  capability,  and 
demonstrate  integrated  system  design  through  developer  testing  to 
show  that  the  system  can  function  in  its  target  environment.  To  enter 
this  phase,  a  program  must  have  approved  requirements  and  pass 
milestone  B.  To  exit  this  phase,  the  acquisition  must  meet  performance 
requirements  in  the  intended  environment. 


sFederal  Acquisition  Regulation,  Part  16.505  (b)  (2)  i-iii. 

9The  defense  acquisition  system  is  a  framework-based  approach  that  is  intended  to 
translate  mission  needs  and  requirements  into  stable,  affordable,  and  well-managed 
acquisition  programs. 
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4.  Production  and  deployment:  The  purpose  of  this  phase  is  to  achieve 
an  operational  capability  that  satisfies  the  mission  needs,  as  verified 
through  independent  operational  test  and  evaluation,  and  to 
implement  the  system  at  all  applicable  locations.  To  enter  this  phase,  a 
program  must  have  completed  development  testing  and  pass  milestone 
C.  To  exit  this  phase,  the  system  must  be  deployed  and  ready  to 
operate  for  all  users. 

5.  Operations  and  support:  The  purpose  of  this  phase  is  to 
operationally  sustain  the  system  in  the  most  cost-effective  manner 
over  its  life  cycle.  DOD  criteria  do  not  require  that  the  milestone 
decision  authority  conduct  milestone  reviews  during  the  period  after  a 
system  has  been  deployed  and  stabilized. 

For  the  purpose  of  conducting  milestone  reviews,  AHLTA  was  assigned 
the  highest  level  of  oversight  for  DOD  information  system  acquisitions. 10 
As  such,  oversight  was  provided  within  the  Office  of  the  Secretary  of 
Defense. 


Management  Structure  for  Various  DOD  units  were  involved  in  acquiring  and  deploying  AHLTA.  As 
AHLTA  the  principal  advisor  to  the  Assistant  Secretary  of  Defense  for  Health 

Affairs  and  to  the  DOD  medical  leaders  on  all  matters  related  to 
information  management  and  information  technology,  the  MHS  chief 
information  officer  (CIO)  has  primary  responsibility  for  overseeing  the 
acquisition,  development,  testing,  and  deployment  of  AHLTA  to  the 
military  treatment  facilities.  Key  offices  within  the  Office  of  the  MHS  CIO 
perform  critical  information  management  and  information  technology 
functions  to  support  AHLTA,  including  the  Joint  Medical  Information 
Systems  Office,  which  is  responsible  for  the  testing,  implementation, 
training,  fielding  of  system  components,  operations,  maintenance,  and 
ultimate  disposal  of  system  components. 

Also  within  MHS,  the  Composite  Health  Care  System  (CHCS)  II  Program 
Office  was  established  in  January  1997  to  provide  direct  management  of 
the  project;  it  had  operational  responsibility  for  the  acquisition  and 
deployment  of  the  electronic  health  record,  as  well  as  the  migration  of  the 


10 AHLTA  is  assigned  acquisition  category  LAM,  which  is  the  highest  information  system 
acquisition  category  for  IT  Systems  and  is  assigned  to  acquisitions  with  at  least  $126 
million  in  fiscal  year  2000  constant  dollars  in  development  and  deployment  costs  or  at  least 
$378  million  in  fiscal  year  2000  constant  dollars  for  all  system  costs. 
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numerous  standalone  clinical  information  systems.  In  fiscal  year  2000,  the 
CHCS  II  program  office  was  renamed  the  Clinical  Information  Technology 
Program  Office  (CITPO).  In  2008,  with  the  merger  of  CITPO  and  the  MHS 
Theater  Medical  Information  Program  Office — -Joint,  the  office  is  now 
called  the  Defense  Health  Information  Management  System  (DHIMS). 

To  provide  oversight  in  accordance  with  DOD’s  defense  acquisition 
system,  the  Assistant  Secretary  of  Defense  for  Networks  and  Information 
Integration,  within  the  Office  of  the  Secretary  of  Defense,  was  designated 
the  milestone  decision  authority  responsible  for  deciding  at  each 
acquisition  cycle  milestone  whether  the  project  could  proceed  to  the  next 
milestone.  The  project  also  received  oversight  from  several  other  bodies, 
including  the  Human  Resources  Management  Investment  Review  Board, 
headed  by  the  MHS  CIO,  and  the  Overarching  Integrated  Project  Team, 
which  evaluated  project  performance  in  accordance  with  DOD  5000  and 
approved  acquisition  program  baselines  and  acquisition  decision 
memorandums. 

Table  1  summarizes  the  assignment  of  responsibilities  for  AHLTA  among 
the  various  DOD  units. 


Table  1 :  Organizations  Responsible  for  Managing  and  Providing  Oversight  of  AHLTA 

Organization 

Description 

Management  organizations 

Office  of  the  Assistant  Secretary  for  Health  Affairs 

Responsible  for  the  department's  military  health  system  program. 
Establishes  policy  regarding  health  care  operations  and  maintenance. 
Several  units  within  this  office,  including  MHS,  are  involved  in 
acquiring  and  deploying  AHLTA. 

MHS  CIO 

Oversees  the  MHS  information  management  and  technology  program. 

J  oint  Requirements  Oversight  Council 

Approves  mission  need  and  operational  requirements  for  automated 
information  systems  with  joint  (i.e.,  multiservice)  interest. 

J  oint  Medical  Information  Systems  Office— Deputy  CIO 

Supports  health  care  operations  through  design,  development,  test, 
evaluation,  and  deployment  of  medical  information  systems.  The 
Program  Executive  Office  is  responsible  for  each  of  the  program 
management  offices  that  oversee  this  activity. 
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Organization 

Description 

Defense  Health  Information  Management  System  Program 
Office 

Manages  the  acquisition,  development,  deployment,  and  maintenance 
of  AHLTA  and  other  related  systems.  The  program  office  reports  to  the 

J  oint  Medical  Information  Systems  Deputy  CIO.  Within  the  program 
office,  the  project  officer  is  responsible  for  ensuring  successful 
planning,  technical  development,  and  acquisition  of  specific 
information  applications  and  elements  of  AHLTA.  The  office  was 
established  in  J  une  2008  with  the  merger  of  CITPO  — the  original 

CHCS  II  program  office— and  the  Theater  Medical  Information 

Program  (the  office  responsible  for  acquiring  the  theater  portion  of  the 
electronic  health  record). 

Oversight  organizations 

Office  of  the  Assistant  Secretary  of  Defense,  Networks  and 
Information  Integration 

Acts  as  the  milestone  decision  authority  that  authorizes  AHLTA's 
readiness  to  move  into  each  phase  of  the  acquisition  life  cycle,  based 
on  successful  completion  of  the  criteria  for  the  preceding  phase. 
Conducts  milestone  reviews  and  prepares  decision  memorandums. 

Human  Resources  Management  Investment  Review  Board 

This  board  is  responsible  for  annual  certification  to  ensure  AHLTA 
meets  specified  requirements  and  should  be  approved  for  funding. 

AHLTA  Overarching  Integrated  ProductTeam 

Reviews  program  planning  in  support  of  the  milestone  decision 
authority,  including  oversight,  review,  and  evaluation  of  project 
execution  performance  relative  to  DOD  guidance. 

Source:  GAO  analysis  of  DOD  data. 

DOD’s  Inspector  General  and  we  have  previously  reported  on  the 
department’s  actions  toward  acquiring  its  new  health  care  information 
system  and  have  noted  the  need  for  improvement  in  key  management 
areas,  such  as  project  management,  contract  management,  and  risk 
management. 

In  reporting  on  the  department’s  efforts  in  January  1999, 11  the  Inspector 
General  noted  that  the  project  management  system  for  the  acquisition 
(called  CHCS  II  at  the  time  of  the  report)  was  not  complete.  While  finding 
that  DOD  had  taken  positive  actions  to  manage  the  acquisition,  the  report 
noted  that  the  department  had  not  established  a  project  management 
control  system  to  evaluate  and  measure  the  program’s  performance.  In 
addition,  the  report  stated  that  the  program’s  funding  visibility  was  limited 
because  DOD  was  combining  funding  for  sustaining  the  system  with 
modernization  funding  for  CHCS  and  other  clinical  business  area 
automated  systems.  The  Inspector  General  made  recommendations 


Previous  Reviews  of  DOD’s 
Electronic  Health  Record 
Initiatives  Highlighted 
Management  Deficiencies 
and  Risks 


11  DOD  Office  of  the  Inspector  General,  Acquisition  Management  of  the  Composi  te  Health 
Care  II  Automated,  Information  System,  report  number  99-068  (January  21,  1999). 
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related  to  designing  and  implementing  a  project  management  control 
system,  the  reporting  of  funding  for  the  system,  and  providing  milestone 
exit  criteria  that  demonstrated  the  level  of  performance,  accomplishments, 
and  progression. 

Further,  in  May  2006, 12  the  Inspector  General  conducted  an  evaluation  of 
the  project’s  program  requirements,  the  related  acquisition  strategy,  and 
system  testing  to  determine  whether  the  system  was  being  implemented  to 
meet  cost,  schedule,  and  performance  requirements.  While  the  report 
found  that  the  program  management  office  was  using  risk  mitigation 
techniques,  such  as  risk  management,  lessons  learned,  and  performance 
monitoring,  the  program  remained  at  high  risk  because  of  the  complexities 
of  integrating  commercial,  off-the-shelf  software  into  the  existing  program. 
In  particular,  the  report  noted  that  the  program  office  had  not  identified 
any  mitigation  strategies  to  reduce  and  control  program  risk  related  to 
integration  of  commercial,  off-the-shelf  software  for  the  third  block  of 
functionality.  As  a  result,  the  Inspector  General  concluded  that  the 
program  was  vulnerable  to  continued  increases  in  cost,  extended 
schedules  for  implementation,  and  unrealized  goals  in  performance  from 
underestimating  the  difficulties  of  integrating  commercial,  off-the-shelf 
products.  Subsequently,  the  program  office  developed  mitigation 
strategies,  but  the  Inspector  General  reported  that  they  were  inadequate 
and  did  not  follow  risk  management  guidance,  including  identifying 
significant  activities  and  milestones.  Accordingly,  the  Inspector  General 
recommended  that  the  program  office  develop  more  robust  mitigation 
strategies  in  accordance  with  the  program  office’s  risk  management  plan. 

We  have  also  reported  on  DOD’s  management  of  the  system  acquisition, 
noting  the  need  for  improvements.  For  example,  in  2002,  we  reported  that, 
because  the  department  had  not  estimated  the  cost  of  delivering  the  initial 
system  capabilities,  it  had  lacked  a  cost  commitment  against  which  to 
measure  progress.13  In  addition,  we  noted  that  program  benefits  were  in 
question  since  measurements  had  not  yet  begun  and  that  costs  were  about 
two-and-a-half  times  the  1998  estimate.  Further,  DOD  had  initially 
identified  a  single  economic  justification  for  the  entire  project,  which  had 


12DOD  Office  of  the  Inspector  General,  Information  Technology  Management:  Acquisition 
of  the  Armed  Forces  Health  Lonqitudinal  Technology  Application,  report  number  D-2006- 
089  (May  18,  2006). 

13GAO,  Information  Technology:  Greater  Use  of  Best  Practices  Can  Reduce  Risks  in 
Acquiring  Defense  Health  Care  System,  GAO-02-345  (Washington,  D.C.:  Sept.  26,  2002). 


Page  12 


GAO-11-50  DOD’s  Electronic  Health  Record 


been  used  as  the  basis  for  its  system  releases,  and  had  not  treated  the 
releases  as  separate  investment  decisions.  Finally,  DOD  had  not  followed 
performance-based  contracting  practices,  resulting  in  the  risk  that  the 
system  would  take  longer  to  acquire  and  cost  more  than  necessary. 

Accordingly,  we  recommended  that  DOD  expand  its  use  of  best  practices 
in  managing  the  system  by  (1)  modifying  the  project’s  investment  strategy 
to  justify  investment  in  each  system  release  before  beginning  development 
and  measuring  return  on  investment  and  (2)  employing  performance- 
based  contracting  practices  where  possible  on  all  future  delivery  orders. 
The  department  agreed  with  these  recommendations  and  took  actions  to 
update  and  validate  its  life-cycle  cost  estimate  in  September  2002.  This 
was  used  by  the  department  to  approve  the  deployment  of  the  system 
release.  Also,  the  department  employed  performance-based  contracting 
practices,  such  as  using  performance  standards,  quality  assurance  plans, 
and  contractor  incentives  on  CHCS  II  delivery  orders. 


AHLTA  Has  Limited 
Capabilities  and 
Continues  to 
Experience 
Performance 
Problems 


Despite  having  obligated  approximately  $2  billion  over  the  13-year  life  of 
its  initiatives  to  acquire  and  operate  an  electronic  health  record  system,  as 
of  September  2010,  DOD  continued  to  experience  performance  problems 
with  the  one  block  of  AHLTA  functionality  (Block  1)  that  it  had  fully 
deployed  and  with  a  second  block  of  functionality  (Block  2)  that  it  had 
partially  deployed.  Further,  after  having  terminated  its  plans  for  deploying 
the  two  other  blocks  of  functionality  (Block  3  and  Block  4)  that  were 
intended  to  be  part  of  the  system,  the  department  has  identified  April  2011 
as  the  date  by  which  it  now  expects  to  achieve  full  operational  capability 
of  the  scaled-backed  AHLTA  system.  Program  officials  told  us  they  are 
taking  steps  to  stabilize  the  existing  system  capabilities  through  2015,  as 
the  department  proceeds  with  plans  to  pursue  yet  another  new  electronic 
health  record  system. 

In  deploying  Block  1,  the  department  reported  that  it  achieved  all  of  the 
planned  outpatient  capabilities  for  direct  patient  care,  including  encounter 
documentation,  order  entry  and  results  retrieval,  encounter  coding 
support,  consult  tracking,  and  alerts  and  reminders.  According  to  the 
department,  it  deployed  the  AHLTA  outpatient  documentation  capability 
worldwide,  providing  77,000  clinicians  with  the  ability  to  document  over 
148,000  outpatient  encounters  daily.  The  department  stated  that  medical 
providers  can  access  the  patient’s  computer-based  record  from  any 
military  treatment  facility.  Also,  DOD  currently  shares  a  significant 
amount  of  patient  information  with  the  Department  of  Veterans  Affairs, 
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including  outpatient  pharmacy  data,  laboratory  results,  and  radiology 
results  on  shared  and  separated  service  members. 


In  addition,  with  the  deployment  of  Block  2,  including  enhancements  to 
Block  1,  dental  capabilities  were  provided  to  73  of  375  dental  treatment 
facilities,  allowing  graphical  dental  charting,  order  and  entry  results 
retrieval,  and  automated  dental  readiness  classification.  In  this  regard,  the 
capabilities  were  deployed  to  46  Air  Force  dental  medical  facilities,  25 
Navy  facilities,  and  2  Army  facilities.  Further,  program  officials  stated  that 
in  October  2009,  because  of  technical  and  functionality  upgrades  made 
over  time  to  the  legacy  Spectacle  Request  Transmission  System,  funding 
was  ceased  for  optometric  capabilities  for  Block  2.  The  department  stated 
that  it  plans  to  achieve  full  operational  capabilities  by  April  2011.  Table  2 
shows  the  capabilities  planned  and  delivered  for  Blocks  1  and  2. 


Table  2:  Capabilities  Planned  and  Delivered  for  Blocks  1  and  2 

Capability 

Status 

Block  1  (outpatient  care) 

Encounter  documentation 

Met 

Order  entry  and  results  retrieval 

Met 

Encounter  coding  support 

Met 

Consult  tracking 

Met 

Alerts  and  reminders 

Met 

Health  data  dictionary 

Met 

Master  patient  index 

Met 

Role-based  security 

Met 

Block  2 

Dental  charting  and  documentation 

In  progress 

Optometric  documentation  and  order  entry 

Not  Met 

Source:  GAO  analysis  of  DOD  data. 


Nonetheless,  program  officials,  as  well  as  users  of  the  system, 
acknowledged  that  problems  with  the  system’s  performance  have 
persisted.  During  a  demonstration  of  the  system’s  operation  in  April  2010, 
medical  providers  discussed  problems  with  AHLTA,  including  limitations 
in  its  availability  and  usability.  For  example,  the  providers  participating  in 
the  demonstration  stated  that  it  is  time-consuming  to  document 
encounters  using  AHLTA  because  of  the  time  required  to  enter 
information  and  navigate  through  the  application  screens.  Thus,  they 
sometimes  must  document  portions  of  an  outpatient  encounter  after  the 
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patient  leaves.  In  their  experience,  using  the  system  at  the  time  of  the 
encounter  would  take  attention  away  from  the  patient  for  unacceptable 
periods  of  time.  Also,  they  stated  that  when  system  downtime  occurs, 
providers  can  neither  access  patient  data  nor  electronically  document 
care;  in  these  instances,  medical  notes  are  recorded  manually  and  later 
entered  in  the  system  after  it  returns  to  operation — an  inefficient  process. 

As  noted  in  the  earlier  discussion,  since  fiscal  year  2006  the  department 
has  been  taking  steps  to  address  performance  problems  and  enhance 
existing  system  capabilities. 14  DOD  is  proceeding  with  what  it  refers  to  as 
a  “stabilization  effort”  to  continue  making  improvements  to  the  system  and 
provide  ongoing  capabilities  until  a  new  system  is  acquired.  According  to 
DOD  officials,  the  estimated  cost  of  this  effort  for  fiscal  year  2010  through 
fiscal  year  2015  is  $826.3  million.  The  stabilization  effort  is  expected  to 
improve  the  speed,  availability,  and  usability  of  the  system;  moreover, 
according  to  officials  in  the  Office  of  the  Deputy  Secretary  of  Defense,  the 
stabilization  effort  is  expected  to  allow  the  department  to  meet  its  near- 
term  needs  and  implement  additional  enhancements  to  support  its  future 
system. 


DOD  Has  Initiated 
Planning  Activities  for 
the  EHR  Way  Ahead 


Because  AHLTA  has  consistently  experienced  performance  problems  and 
has  not  delivered  the  full  operational  capabilities  intended,  DOD  has 
initiated  plans  to  develop  a  new  electronic  health  record  system.  This  new 
initiative  is  called  the  Electronic  Health  Record  (EHR)  Way  Ahead.  As 
with  AHLTA,  department  officials  stated  that  the  new  electronic  health 
record  system  is  expected  to  be  a  comprehensive,  real-time  health  record 
for  active  and  retired  service  members,  their  families,  and  other  eligible 
beneficiaries.  They  added  that  the  new  system  is  being  planned  to  address 
the  capability  gaps  and  performance  problems  of  previous  iterations,  and 
to  improve  existing  information  sharing  between  DOD  and  the  Department 
of  Veterans  Affairs  and  expand  information  sharing  to  include  private 
sector  providers. 


Thus  far,  the  department  has  taken  several  steps  to  launch  its  acquisition 
of  the  new  system.  Specifically,  in  February  2010  it  established  the  EHR 
Way  Ahead  Planning  Office  to  identify  options  for  the  future  electronic 
health  record  system.  The  planning  office  currently  resides  within  the 


14This  effort  included  the  AHLTA  3.3  software  release  discussed  above. 
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MHS  Joint  Medical  Information  Systems  Program  Executive  Office  under 
the  Office  of  the  CIO. 

In  May  2010,  the  department  approved  plans  to  assess  solutions  for  the 
new  electronic  health  record  system.  In  this  regard,  the  planning  office 
began  conducting  an  analysis  of  alternatives  to  provide  guidance  on 
selecting  a  technical  solution.  According  to  planning  officials,  efforts  to 
develop  the  analysis  of  alternatives  are  being  supervised  by  the  Office  of 
the  Assistant  Secretary  of  Defense  for  Health  Affairs,  and  this  analysis  is 
expected  to  define  and  evaluate  reasonable  alternatives  for  meeting  the 
capability  requirements.  The  analysis  is  currently  scheduled  to  be 
completed  by  December  2010. 

To  facilitate  the  analysis  of  alternatives,  planning  officials  stated  that  they 
had  identified  system  capabilities  needed  to  meet  the  department’s 
medical  mission.  They  added  that  a  list  of  the  “top  10”  priority  capabilities 
for  a  new  system  had  been  developed  based  on  the  gaps  identified  in  prior 
iterations  of  their  electronic  health  systems.  (These  priorities  are 
summarized  in  table  3.) 


Table  3:  Top  10  Priorities  for  EHR  Way  Ahead 

Priority 

Capability  needed  to  meet  DOD’s  medical  mission 

1 

Comprehensive  medical  and  dental  documentation,  including  encounter  data,  medications,  physical  examinations, 
occupational  health  (including  industrial  hygiene),  environmental  exposure  information  and  ancillary  service  data  (both 
inpatient  and  outpatient),  documentation  of  care  plan  objectives,  alternatives,  patient  education,  health  care  services 
provided,  patient  disposition  instructions  (including  deaths),  and  disposition  of  remains. 

2 

Global  capture  and  exchange  of  all  health  data  for  beneficiaries— direct  care,  network,  managed  care,  Veterans 

Affairs,  active  duty  components,  reserve  components,  etc. 

3 

Inpatient  and  outpatient  order  entry  and  management  (laboratory,  pharmacy,  radiology,  consults,  health  care  plans, 
nutrition  management,  prescription  spectacle  orders). 

4 

Laboratory  diagnostic  services  (includes  results,  retrieval  and  reporting);  pharmacy  services  (includes  dispensing, 
operations,  reporting,  and  pharmacy  data  transaction  service);  radiology  diagnostic  services  (includes  imagery 
capture,  results,  retrieval,  and  reporting). 

5 

En-route  care  documentation  on  any  transport  platform. 

6 

Results  retrieval  (ancillary  services  and  consults). 

7 

Data  collection  and  decision  support  in  austere  environments  starting  at  the  point  of  injury  and  continuing  through  all 
levels  of  care. 

8 

Consult  and  referral  management  (includes  referrals  to  the  civilian  health  care  sector). 

9 

Assessments  of  medical  deployability  of  individual  service  members. 

10 

Patient  administration  (includes  who  the  patient  is,  what  he/she  is  entitled  to,  where  he/she  is  located,  etc.). 

Source:  GAO  analysis  of  DOD  data. 
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According  to  planning  documents,  following  completion  of  the  analysis, 
DOD  expects  to  select  a  technical  solution  and  to  develop  and  release  a 
delivery  schedule. 

DOD’s  fiscal  year  2011  budget  request  includes  $302  million  for  the  EHR 
Way  Ahead  initiative. 15  For  fiscal  year  2012,  the  department  intends  to 
submit  an  updated  budget  request  and  the  schedule  for  delivery  of  the 
EHR  Way  Ahead  based  on  the  results  of  the  analysis  of  alternatives. 


AHLTA  Performance 
Was  Hindered  by 
Weaknesses  in  Key 
Acquisition 
Management  and 
Planning  Processes 


The  success  of  a  large  information  technology  project  such  as  AHLTA  is 
dependent  on  an  agency  possessing  capabilities  to  effectively  plan  and 
manage  acquisitions,  design  the  associated  systems,  define  and  manage 
system  requirements,  and  use  effective  measures  to  gauge  user 
satisfaction.  In  the  case  of  AHLTA,  weaknesses  in  these  key  management 
areas  contributed  to  DOD  delivering  a  system  that  provided  fewer 
capabilities  than  originally  expected,  experienced  persistent  performance 
problems,  and  ultimately,  did  not  fully  meet  the  needs  of  its  intended 
users.  Alleviating  these  areas  of  weakness  will  be  essential  to  the  success 
of  further  initiatives,  including  the  AHLTA  stabilization  effort  and  the  EHR 
Way  Ahead,  that  the  department  undertakes  in  pursuit  of  its  electronic 
health  record  system  capabilities. 


Project  Plan  Was  Program  management  principles  and  best  practices  emphasize  the 

Incomplete  and  Not  importance  of  having  a  project  management  plan  in  place  that,  among 

Maintained  Other  things,  establishes  a  complete  description  that  ties  together  all 

program  activities  and  evolves  over  time  to  continuously  reflect  the 
current  status  and  desired  end  point  of  the  project.16  An  effective  plan  is 
comprised  of  a  description  of  the  program’s  scope,  cost,  lines  of 
responsibility  and  authority,  management  processes,  and  schedule.  Such  a 
plan  incorporates  all  the  critical  areas  of  system  development  and  is  to  be 


15In  addition,  DOD  plans  to  spend  $40  million  on  a  related  effort  to  test  the  exchange  of 
electronic  health  records  with  the  Department  of  Veterans  Affairs  and  private  health  care 
providers  and  to  work  toward  a  goal  announced  by  President  Obama  on  April  9,  2009,  that 
the  departments  would  cooperate  to  create  a  joint  virtual  lifetime  electronic  health  record 
for  service  members  and  veterans. 

16See  Institute  of  Electrical  and  Electronics  Engineers  (IEEE),  IEEE/EIA  Guide  for 
Information  Technology,  IEEE/EIA  12207.1-1997  (April  1998)  and  Carnegie  Mellon 
Software  Engineering  Institute,  Capability  Maturity  Model  Integration  for  Acquisition, 
Version  1.2  (Pittsburgh,  Pa,  November  2007). 
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used  as  a  means  of  determining  what  needs  to  be  done,  by  whom,  and 
when. 

Other  guidance,  such  as  our  Information  Technology  Investment 
Management  framework, 17  states  that  effective  program  oversight  of  IT 
projects  and  systems,  including  those  in  operation  and  maintenance, 
involves  maintaining  approved  project  management  plans  that  include 
expected  cost  and  schedule  milestones  and  measurable  benefit  and  risk 
expectations. 

However,  officials  did  not  follow  best  practices  in  developing  a  project 
management  plan  to  guide  the  department’s  electronic  health  record 
system.  Although  the  department  established  a  project  management  plan, 
it  did  not  include  several  standard  components  such  as  the  project’s 
scope,  a  requirements  management  plan,  cost  estimates  and  baseline,  a 
schedule,  and  a  staffing  management  plan.  In  addition,  although  DOD 
identified  the  plan  as  a  keystone  document  for  guiding  the  project,  the 
plan  was  last  revised  in  2005  and  was  not  updated  during  subsequent 
development  work  and  the  operations  and  maintenance  phase  to  reflect 
significant  changes  to  the  program.  These  changes  included  termination 
and  postponement  of  planned  capabilities,  and  revisions  to  the  acquisition 
processes  used  to  guide  the  AHLTA  program.  As  a  result,  a  plan  was  not  in 
place  to  effectively  guide  the  program  throughout  these  changes. 
Moreover,  there  is  no  such  plan  to  guide  current  activities  associated  with 
the  stabilization  effort,  which,  as  discussed  previously,  involves  attempts 
to  address  system  performance  problems  and  enhance  functionality. 

According  to  program  officials,  the  project  management  plan  was  last 
revised  in  2005  before  their  focus  shifted  to  addressing  the  system 
performance  problems  that  occurred  as  a  result  of  completing  Block  1 
deployment  in  December  2006.  Nevertheless,  significant  changes  occurred 
to  the  program’s  scope,  cost,  and  schedule  after  Block  1  deployment,  and 
the  agency  lacked  a  current  and  complete  plan  to  guide  activities  and 
measure  program  progress.  Going  forward,  developing  and  maintaining  a 
comprehensive  project  plan  will  be  an  essential  tool  for  overseeing  the 
AHLTA  stabilization  effort,  which  is  to  provide  crucial  improvements  to 
the  system  and  act  as  a  bridge  over  the  next  5  years  to  the  deployment  of 
the  EHR  Way  Ahead  system.  Further,  having  a  comprehensive  and  current 


17GAO,  Information  Technology  Investment  Management:  A  Framework  for  Assessing 
and  Improving  Process  Maturity,  GAO-04-394G  (Washington,  D.C.:  March  2004). 
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project  plan  for  the  EHR  Way  Ahead  program  will  help  to  guide  the  project 
and  provide  oversight  of  the  project’s  progress.  Without  a  project 
management  plan  that  reflects  the  status  and  goals  of  the  project,  DOD 
increases  the  risk  that  stakeholders  will  not  have  the  insight  into  program 
status  that  is  needed  to  exercise  effective  oversight  of  both  the  AHLTA 
stabilization  effort  and  the  EHR  Way  Ahead  acquisition. 


DOD  Lacked  a  Systems 
Engineering  Plan  to  Guide 
the  Electronic  Health 
Record  System’s  Design 


According  to  industry  best  practices,18  systems  engineering  governs  the 
total  technical  and  managerial  effort  required  to  transform  a  set  of  user 
requirements  and  expectations  into  specific  capabilities  and,  ultimately, 
into  a  system  design  that  will  meet  users’  needs.  Systems  engineering 
practices  include  developing  solutions  for  achieving  system  performance 
requirements  such  as  system  availability,  and  ensuring  compatibility  when 
integrating  multiple  systems  and  their  components.  Further,  DOD 
guidance  states  that  a  tailored  and  detailed  systems  engineering  plan  is  a 
critical  tool  for  guiding  systems  engineering  practices  throughout  the  life 
of  an  acquisition  program.  Having  such  a  plan  is  particularly  important  for 
a  system  characterized  by  significant  technical  complexities. 


DOD’s  electronic  health  record  system  design  reflected  numerous 
technical  complexities,  such  as  the  need  to  capture,  manage,  and  share 
health  information  across  a  worldwide  network  that  must  be  available  24 
hours  a  day,  7  days  a  week,  and  that  is  to  serve  a  transient  patient 
population.  In  addition,  the  system  design  involved  a  network  that  had  to 
be  integrated  with  a  central  patient  database  and  multiple  nonstandard 
hardware  and  software  platforms,  such  as  commercial,  off-the-shelf 
products  at  over  800  military  treatment  facilities. 


Nonetheless,  although  the  program  office  recognized  these  types  of  system 
complexities  as  being  part  of  the  electronic  health  record  system  design, 
the  office  never  established  a  tailored  systems  engineering  plan  to  guide 
the  acquisition,  or  to  facilitate  the  resolution  of  the  many  performance 
problems  that  have  plagued  the  system  since  its  initial  deployment. 

In  this  regard,  a  particularly  troublesome  area  for  the  department  has  been 
in  deploying  enhancements  to  the  system.  For  example,  following  Block  1 
deployment  in  2006,  the  department  implemented  local  cache  servers  in  an 


18Carnegie  Mellon  Software  Engineering  Institute,  Capability  Maturity  Model  Integration 
for  Acquisi  tion,  Version  1.2. 
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attempt  to  improve  the  system’s  operational  availability.  According  to  the 
department,  the  specific  purpose  of  the  local  cache  servers  had  been  to 
mitigate  the  need  to  access  patient  medical  information  in  the  central  data 
repository  during  system  outages.  However,  after  the  servers  were 
deployed,  DOD  realized  that  the  placement  of  the  servers  within  the 
system  architecture  did  not  resolve  the  problem  and  created  a  single  point 
of  failure.  Rather  than  yield  operational  improvements,  department 
officials  acknowledged  that  these  actions  resulted  in  additional  challenges, 
including  the  need  for  a  costly  local  cache  server  redesign,  which  was 
begun  in  fiscal  year  2009.  Program  documentation  noted  that  the  local 
cache  server  effort  was  probably  one  of  the  most  difficult  engineering 
challenges  that  the  program  office  had  faced  so  far.  Further,  as  various 
issues  were  faced,  it  became  increasingly  clear  that  detailed  planning  in 
the  earlier  stages  was  not  what  it  could  have  been.  In  April  2010,  clinicians 
demonstrating  the  system  at  the  Bethesda  Naval  Medical  Center  stated 
that  the  servers  continued  to  be  a  major  contributing  factor  to  system 
availability  issues. 19 

The  lack  of  a  systems  engineering  plan  to  guide  the  program  office 
through  this  type  of  complexity  is  particularly  notable  in  light  of  the  DOD 
Inspector  General’s  report  of  2006,  which  stated  that  inadequate  planning 
for  technical  complexities  significantly  impacts  the  cost,  schedule,  and 
performance  of  a  program.  The  report  further  stated  that  the  AHLTA 
program  office  had  underestimated  the  technical  complexity  of  integrating 
products  with  the  electronic  health  record  system  and,  as  a  result, 
remained  at  high  risk  for  continued  cost  increases,  schedule  overruns,  and 
unrealized  performance  goals. 

In  discussing  this  matter,  agency  officials  stated  that  a  tailored  systems 
engineering  plan  had  not  been  developed  to  guide  the  design  of  AHLTA 
because  such  a  plan  was  not  required  when  the  system  was  originally 


19 According  to  DOD,  the  desirable  target  for  AHLTA  system  availability  is  100  percent, 
meaning  that  the  system  is  available  to  users  whenever  it  is  needed,  and  the  performance 
threshold  is  99  percent,  meaning  that  if  availability  falls  below  99  percent,  performance  is 
considered  to  be  unacceptable.  Further,  a  system  performance  report  for  the  time  period 
October  2008  to  February  2010  did  not  show  any  months  with  availability  at  the  desired 
level  of  100  percent,  and  only  1  month  when  it  was  available  at  the  acceptable  level 
between  99  and  100  percent,  and  then  only  at  the  Army  and  Navy  facilities.  The  Air  Force 
experienced  the  lowest  levels  of  availability,  with  7  months  that  were  between  93  and  97 
percent  availability.  The  report  showed  that  system  downtime  included  some  system 
maintenance,  but  the  primary  cause  of  downtime  was  implementation  of  improvements  to 
address  performance  problems. 
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planned.  Specifically,  the  officials  stated  that,  it  was  not  until  February 
2004  that  DOD  issued  a  policy  requiring  that  a  systems  engineering  plan  be 
in  place  for  acquisition  programs’  milestone  reviews;  but  all  milestone 
reviews  for  AHLTA  had  been  completed  prior  to  this  time. 

However,  current  DOD  guidance  emphasizes  the  need  for  a  tailored 
systems  engineering  plan  to  guide  all  systems  engineering  practices, 
including  those  that  occur  after  the  completion  of  milestone  reviews. 
Without  a  tailored  systems  engineering  plan  to  guide  the  program’s  efforts 
to  address  long-standing  system  performance  problems  as  part  of  the 
AHLTA  stabilization  efforts,  the  department  may  continue  to  be  challenged 
in  achieving  the  desired  results.  Further,  in  planning  for  the  acquisition  of 
the  new  EHR  Way  Ahead  system,  it  will  be  essential  that  the  department 
establish  early  in  the  process  and  have  in  place  a  detailed  and  tailored  plan 
to  avoid  encountering  technical  challenges  similar  to  those  of  the  AHLTA 
program,  and  thus  again  failing  to  meet  users’  needs. 


Weaknesses  in  DOD’s 
Requirements  Processes 
Impacted  AHLTA’s 
Usability 


According  to  recognized  guidance,20  using  disciplined  processes  for 
developing  and  managing  requirements  can  help  reduce  the  risks  of 
developing  a  system  that  does  not  meet  user  and  operational  needs. 
Requirements  should  serve  as  the  basis  for  establishing  agreement 
between  users  and  developers  and  a  shared  understanding  of  the  system 
to  be  developed.  Effective  requirements  development  practices  include, 
among  other  things,  involving  users  in  identifying  requirements  throughout 
the  project’s  life  cycle  to  ensure  system  requirements  are  complete  and 
accurately  reflect  their  needs.  Effective  requirements  management 
practices  include  maintaining  bidirectional  traceability  of  requirements  to 
ensure  that  system-level  requirements  can  be  traced  both  backward  to 
high-level  operational  requirements,  and  forward  to  low-level  system 
design  specifications. 


For  the  AHLTA  acquisition,  program  documentation  revealed  that  users 
were  not  adequately  involved  throughout  the  requirements  development 
process.  According  to  the  documentation,  users  did  not  seek  involvement 
in  the  requirements  development  process  and  system  developers  did  not 
seek  user  input  when  making  changes  to  requirements.  As  a  result, 


20See  Federal  Acquisition  Regulation  39.102  and  Carnegie  Mellon,  Software  Engineering 
Institute,  Capability  Maturity  Model-Integration  for  Development,  Version  1.2 
(Pittsburgh,  Pa.,  August  2006)  and  Software  Acquisition  Capability  Maturity  Model, 
Version  1.03,  CMU/SEI-2002-TR-010  (Pittsburgh,  Pa.,  March  2002). 
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requirements  were  neither  complete  nor  sufficiently  detailed  to  guide 
system  development,  and  did  not  adequately  provide  a  shared 
understanding  between  the  users  and  developers  of  how  the  system  was  to 
be  developed.  Program  documentation  noted  that  requirements  often  were 
not  adequately  specified  and  did  not  adequately  reflect  user  needs.  In 
particular,  the  program  documentation  revealed  that,  while  users  were 
involved  in  developing  an  initial  set  of  requirements  used  to  make  system 
acquisition  decisions,  they  were  largely  not  involved  in  identifying  new 
requirements  and  making  changes  to  existing  ones  while  the  system  was 
being  developed  and  deployed. 

In  certain  instances,  because  users  were  involved  only  at  the  beginning 
and  end  of  the  requirements  development  process,  they  were  only  able  to 
determine  that  capabilities  would  not  meet  their  needs  after  those 
capabilities  had  already  been  deployed.  For  example,  when  the  dental 
application  was  in  the  process  of  being  deployed  to  Army,  Navy,  and  Air 
Force  sites,  the  MHS  senior  leadership  voted  to  halt  further  training  and 
implementation  because  users  reported  that  the  capabilities  were  not 
complete  and  did  not  address  their  needs.  Consequently,  alternate  dental 
solutions  will  be  explored  as  part  of  the  analysis  of  alternatives  for  the 
EHR  Way  Ahead,  resulting  in  additional  costs  and  delays  in  deploying 
dental  capabilities  that  will  meet  users’  requirements. 

Since  the  initial  deployment,  the  department  has  taken  steps  to  increase 
user  involvement  in  defining  requirements.  For  example,  to  better  involve 
users  in  the  requirements  process  and  identify  issues  with  system 
usability,  the  program  office  held  conferences  in  2006  at  which  users 
identified  over  200  new  requirements  for  inclusion  in  the  system.  Program 
officials  stated  that  the  requirements  identified  during  the  conference 
were  used  to  develop  the  AHLTA  3.3  software  release.  However,  our 
evaluation  of  the  requirements  traceability  matrix  used  to  develop  the 
AHLTA  3.3  release  showed  that  bidirectional  traceability  had  not  been 
fully  established;  thus,  the  requirements  were  not  always  linked  to  high- 
level  operational  requirements  or  to  more  detailed  design  specifications. 
Without  adequate  traceability,  the  department  cannot  ensure  that  all 
agreed-upon  requirements  will  be  developed,  fully  tested,  and  work  as 
intended. 

In  addition,  the  department  has  plans  for  making  improvements  in  the 
requirements  management  process  in  its  MHS  Information 
Management/Information  Technology  Strategic  Plan  2010-2015  and 
includes  a  goal  to  improve  the  requirements  management  process  to 
enable  greater  participation  of  system  users.  According  to  the  plan,  this 
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will  improve  the  value,  quality,  timeliness,  and  stakeholder  ownership  of 
the  resulting  system.  However,  because  the  department  is  in  the  early 
stages  of  implementing  improvements  for  greater  user  participation,  it  is 
too  early  to  determine  their  effectiveness. 

As  the  department  proceeds  with  the  AHLTA  stabilization  effort  and  the 
new  EHR  Way  Ahead  system,  ensuring  that  user  needs  are  met  will  be 
essential  to  effective  and  cost-efficient  delivery  of  system  capabilities. 
Until  the  department  ensures  that  a  requirements  development  process 
with  adequate  user  involvement  is  in  place,  it  will  continue  to  lack  a  vital 
tool  for  ensuring  the  efficient  and  effective  delivery  of  electronic  health 
record  system  capabilities  that  will  meet  the  needs  of  its  users. 


Efforts  to  Improve  User 
Satisfaction  Were  Not 
Guided  by  Effective 
Planning 


DOD  has  stated  that  the  success  of  AHLTA  can  be  gauged  by 
improvements  in  user  satisfaction  and  user  acceptance,  among  other 
things.  In  this  regard,  effectively  managing  program  improvement 
activities  to  improve  user  satisfaction  requires  planning  and  executing 
such  activities  in  a  disciplined  fashion.  The  Software  Engineering 
Institute’s  IDEAL™21  model  is  a  recognized  approach  for  managing  efforts 
to  make  system  improvements.  According  to  this  model,  user  satisfaction 
improvement  efforts  should  include  a  written  plan  that  serves  as  the 
foundation  and  basis  for  guiding  improvement  activities,  including 
obtaining  management  commitment  to  and  funding  for  the  activities, 
establishing  a  baseline  of  commitments  and  expectations  against  which  to 
measure  progress,  prioritizing  and  executing  activities  and  initiatives, 
determining  success,  and  identifying  and  applying  lessons  learned. 
Through  such  a  structured  and  disciplined  approach,  improvement 
resources  can  be  invested  in  a  manner  that  produces  optimal  results. 

However,  DOD  has  not  demonstrated  that  user  satisfaction  improvement 
efforts  are  being  guided  by  a  documented  plan  that  defines  prioritized 
improvement  projects  and  associated  resource  requirements,  schedules, 
and  measurable  goals  and  outcomes.  Instead,  efforts  that  the  office 
undertook  to  improve  user  satisfaction  were  ad  hoc  and  did  not  meet  with 
success.  Specifically,  the  program  office  stopped  measuring  AHLTA  user 


21The  Software  Engineering  Institute  is  a  federally  funded  research  and  development  center 
established  at  Carnegie  Mellon  University  to  address  software  engineering  practices. 
IDEAL™  is  a  service  mark  of  Carnegie  Mellon  University  and  stands  for  initiating, 
diagnosing,  establishing,  acting,  and  leveraging.  For  more  information  on  this  model,  see 
IDEALSU :  A  User’s  Guide  for  Software  Process  Improvement  (CMU/SEI-96-HB-001). 
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satisfaction  levels  in  July  2007  after  overall  user  satisfaction  had  declined 
to  its  lowest  point  in  more  than  2  years.  Between  2005  and  2007  the 
program  office  collected  user  satisfaction  feedback  through  online  user 
surveys,  and  used  the  data  to  identify  areas  for  system  improvements  and 
to  measure  progress  toward  improving  satisfaction.  The  results  of  the 
surveys  showed  not  only  that  users  rated  their  overall  satisfaction  level 
with  the  system  between  below  average  and  average,  but  that  user 
satisfaction  levels  had  declined  to  a  low  point  with  the  results  of  the  final 
survey  report  of  July  2007.  Thus,  as  shown  in  figure  1,  the  program  office 
was  not  able  to  improve  user  satisfaction  during  this  time  period. 


Figure  1 :  Overall  AHLTA  User  Satisfaction  Ratings  between  April  2005  and  July 
2007 


According  to  program  officials,  they  have  implemented  a  major  effort 
toward  improving  user  satisfaction  with  the  AHLTA  3.3  software  release. 
The  improvements  associated  with  this  software  release  began  as  early  as 
2006  and  include  features  such  as  improved  medical  coding  support  and 
increased  speed  of  the  order  entry  connection,  as  well  as  other  changes  to 
improve  users’  satisfaction  with  the  system’s  performance  and 
capabilities.  Yet,  program  officials  did  not  provide  evidence  of  a  plan  to 
guide  these  efforts  or  a  schedule  for  implementing  these  improvements, 
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and  it  is  unclear  how  specific  capabilities  of  the  software  release  will  be 
used  to  address  specific  user  concerns.  The  lack  of  a  documented  plan  to 
guide  user  satisfaction  improvement  activities  is  of  particular  significance 
because  users  have  continued  to  express  their  dissatisfaction  with  the 
system.  Program  officials  stated  that  additional  online  user  satisfaction 
surveys  were  not  conducted  after  2007  because  users  had  grown  weary  of 
the  surveys  and  efforts  to  address  user  feedback  from  the  existing  survey 
results  are  ongoing.  The  next  online  survey  is  expected  to  be  conducted 
after  full  deployment  of  AHLTA  3.3,  but  a  schedule  has  not  yet  been  set. 

Given  the  history  of  system  performance  problems  and  the  extent  to 
which  users  have  not  been  able  to  effectively  and  efficiently  use  AHLTA,  it 
is  critical  that  the  department  identify  and  implement  system 
improvements  in  a  disciplined  and  structured  fashion.  Without  a 
documented  improvement  plan,  efforts  to  improve  user  satisfaction, 
including  those  associated  with  the  ongoing  AHLTA  stabilization  effort, 
may  be  reduced  to  trial  and  error,  and  the  office  cannot  adequately  ensure 
that  it  is  effectively  investing  program  resources  on  improvement  efforts 
that  will  result  in  a  system  that  satisfies  users.  Further,  since  increasing 
user  satisfaction  is  a  key  goal  for  the  EHR  Way  Ahead,  it  is  critical  that  a 
disciplined  approach  is  established  and  maintained  throughout  the 
program’s  life  cycle. 


MHS  Lacks  Assurance  of  a 
Disciplined  Acquisition 
Management  Process  to 
Guide  Its  Electronic  Health 
Record  Initiative 


The  use  of  disciplined  processes  to  guide  the  effort  of  acquiring  and 
implementing  a  major  system  has  been  shown  to  increase  the  likelihood  of 
achieving  intended  results  and  reduce  the  risks  associated  with  an 
acquisition  to  acceptable  levels.  Although  there  is  no  standard  set  of 
practices  that  will  ever  guarantee  success,  several  organizations,  such  as 
Carnegie  Mellon  University’s  Software  Engineering  Institute  and  the 
Institute  of  Electrical  and  Electronics  Engineers  (IEEE),22  as  well  as 
individual  experts,  have  identified  and  developed  the  types  of  policies, 
procedures,  and  practices  that  have  been  demonstrated  to  reduce 
development  time  and  enhance  effectiveness.  The  key  to  having  a 
disciplined  system  development  effort  is  to  have  disciplined  processes  in 
multiple  areas,  including  project  planning,  requirements  management, 
systems  engineering,  system  testing,  and  risk  management.  Because 


22The  IEEE  is  a  nonprofit,  technical  professional  association  that  develops  standards  for  a 
broad  range  of  global  industries,  including  the  IT  and  information  assurance  industries  and 
is  a  leading  source  for  defining  best  practices. 
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change  in  a  program  is  constant,  effective  processes  should  be 
implemented  in  each  of  these  throughout  the  project  life  cycle.  Effectively 
implementing  the  disciplined  processes  necessary  to  reduce  project  risks 
to  acceptable  levels  is  difficult  because  a  project  must  effectively 
implement  several  best  practices,  and  inadequate  implementation  of  any 
one  may  significantly  reduce  or  even  eliminate  the  positive  benefits  of  the 
others. 

Recognizing  weaknesses  in  its  acquisition  of  systems  such  as  AHLTA,  MHS 
has  been  taking  steps  to  institutionalize  more  disciplined  management 
processes  across  all  of  its  programs.  In  March  2008  the  MHS  CIO  identified 
an  approach  for  improving  its  management  processes  that  included 
aligning  MHS  processes  with  best  practices  outlined  in  the  Software 
Engineering  Institute’s  Capability  Maturity  Model  Integration  (CMMI)  for 
Acquisition.  In  support  of  the  approach,  certain  program  offices,  including 
DHIMS  (the  program  office  responsible  for  the  AHLTA  acquisition),  were 
selected  for  an  internal  evaluation  to  identify  areas  for  improvement  in  the 
existing  MHS  processes.  The  assessment,  which  was  conducted  in  May 
2008,  identified  weaknesses  in  processes  such  as  project  management, 
requirements  development,  and  project  monitoring  and  control,  among 
others.  It  also  identified  weaknesses  in  MHS’s  oversight  of  the 
implementation  of  these  processes  within  program  offices.  Specifically, 
the  assessment  identified  weaknesses  in  the  area  of  Process  and  Product 
Quality  Assurance,  which  is  supposed  to  provide  staff  and  management 
with  objective  insight  into  processes  associated  with  work  products.  The 
assessment  found  little  evidence  that  process  evaluations  were  performed 
across  the  organization,  quality  assurance  audits  were  conducted,  and 
noncompliance  issues  were  tracked  and  reported. 

In  response  to  the  assessment,  officials  stated  that  they  established  a  plan 
for  addressing  the  identified  weaknesses.  Specifically,  their  goal  was  to 
achieve  CMMI’s  “maturity  level  2”  for  processes  such  as  project  planning 
and  acquisition  requirements  development.  Level  2  processes  are 
“managed”  processes,  or  processes  that  are  planned  and  executed  in 
accordance  with  policy;  employ  skilled  people  who  have  adequate 
resources  to  produce  controlled  outputs;  involve  relevant  stakeholders; 
are  monitored,  controlled,  and  reviewed;  and  are  evaluated  for  adherence 
to  their  process  description.  The  department  planned  to  conduct  a  formal 
external  assessment  of  the  maturity  of  its  processes  by  December  2008. 

Program  officials  stated  that  they  provided  guidance  and  assistance  for 
program  offices  to  adopt  practices  associated  with  CMMI  maturity  level  2 
processes.  However,  they  have  yet  to  perform  the  planned  external 
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assessment  of  their  processes,  and  there  is  therefore  little  assurance  that 
improvements  have  been  carried  out.  As  the  department  proceeds  with  the 
AHLTA  stabilization  effort,  it  is  critical  that  it  have  disciplined  processes  in 
place  to  avoid  past  problems  with  not  delivering  system  improvements  as 
planned.  Further,  as  the  department  is  allocating  resources  to  and 
planning  for  the  EHR  Way  Ahead  acquisition,  it  is  critical  that  it  have 
disciplined  management  processes  in  place  to  avoid  repeating  the 
mistakes  of  the  past.  Until  the  department  ensures  that  these  disciplined 
and  managed  processes  are  in  place,  it  risks  delivering  another  system 
with  limited  functionality  and  performance  problems  and  that  does  not 
meet  the  needs  of  its  users. 


Conclusions 


After  over  a  decade  of  effort,  DOD  has  not  accomplished  what  it  set  out  to 
achieve  in  acquiring  a  comprehensive  electronic  health  record  system. 
While  it  has  delivered  a  number  of  outpatient  capabilities,  weaknesses  in 
key  management  areas  hindered  its  ability  to  deliver  the  full  complement 
of  intended  capabilities  and  to  ensure  that  the  capabilities  it  has  delivered 
meet  required  performance  parameters.  The  program  office  did  not 
maintain  a  comprehensive  and  current  project  management  plan,  a  critical 
document  that  provides  stakeholders  insight  into  the  project’s  plans  and 
status.  Also,  despite  the  department’s  need  to  deliver  a  complex, 
worldwide  system,  it  did  not  develop  a  systems  engineering  plan  to  help 
address  the  technical  aspects  of  the  project,  and  it  continues  to  experience 
problems  with  system  availability,  speed,  and  usability.  Further,  the 
system  requirements  were  too  general  and  did  not  adequately  reflect  user 
needs.  Although  the  department  has  collected  user  feedback,  it  did  not 
establish  a  comprehensive  plan  for  improving  user  satisfaction  with  the 
system.  Recognizing  weaknesses  in  acquisition  management  areas,  the 
MHS  CIO  issued  guidance  for  improving  its  management  processes,  but  it 
has  not  performed  the  planned  external  assessment  that  it  needs  to  certify 
that  these  improvements  have  been  made  or  established  a  date  for  doing 
so. 


As  DOD  continues  to  invest  significant  resources  in  a  stabilization  effort  to 
address  shortcomings  of  AHLTA  and  plan  for  the  acquisition  of  a  new 
electronic  health  record  system,  it  is  imperative  that  the  department  take 
immediate  steps  to  improve  its  management  of  the  initiative.  Until  it  does 
so,  it  risks  a  continuation  of  the  problems  it  has  already  experienced, 
which  could  again  prevent  DOD  from  delivering  a  comprehensive  health 
record  system  for  serving  its  service  members  and  their  families. 
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Recommendations  for 
Executive  Action 


To  help  guide  and  ensure  the  successful  completion  of  the  AHLTA 
stabilization  effort,  we  recommend  that  the  Secretary  of  Defense,  through 
the  Assistant  Secretary  of  Defense  for  Health  Affairs,  direct  the  MHS  CIO 
to  take  the  following  six  actions: 


•  Develop  and  maintain  a  comprehensive  project  plan  that  includes  key 
elements,  such  as  the  project’s  scope,  cost,  schedule,  and  risks  and  update 
the  plan  to  provide  key  information  for  stakeholders  on  the  project’s  plans 
and  status. 


•  Develop  a  systems  engineering  plan  in  accordance  with  DOD  guidance  to 
address  the  technical  complexities  of  delivering  a  worldwide  electronic 
health  record  system. 

•  Ensure  that  its  requirements  development  process  involves  system  users 
throughout  the  development  process,  to  obtain  an  understanding  of  what 
will  satisfy  their  needs. 

•  Ensure  the  establishment  of  bidirectional  traceability  for  all  system 
requirements. 

•  Develop  and  document  a  plan  for  improving  user  satisfaction  that 
prioritizes  improvement  projects;  identifies  needed  resources;  includes 
schedules  for  improvement  efforts,  including  future  user  feedback 
surveys;  and  links  efforts  to  measurable  outcomes  and  specific  user  needs. 

•  Establish  acquisition  management  processes  in  accordance  with  industry 
best  practices,  including  identifying  milestones  and  a  completion  date  for 
the  external  evaluation  that  MHS’s  processes  are  at  maturity  level  2  of  the 
Capability  Maturity  Model  Integration  for  Acquisition. 

Further,  to  help  ensure  that  the  EHR  Way  Ahead  does  not  have  shortfalls 
similar  to  those  experienced  with  AHLTA,  we  recommend  that  the  above 
six  management  practices  be  implemented  as  part  of  the  planning  for  this 
important  initiative. 


Agency  Comments 
and  Our  Evaluation 


The  Deputy  Assistant  Secretary  of  Defense  (Force  Health  Protection  and 
Readiness),  performing  the  duties  of  the  Assistant  Secretary  of  Defense 
(Health  Affairs),  provided  written  comments  on  a  draft  of  this  report.  In  its 
comments,  the  department  agreed  with  our  six  recommendations  and 
described  actions  planned  to  address  them. 
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Specifically,  to  help  guide  and  ensure  the  successful  completion  of  the 
AHLTA  stabilization  effort,  DOD  stated  that  it  will  develop  and  maintain  a 
comprehensive  project  plan  in  accordance  with  our  recommendation  and 
DOD  acquisition  program  guidelines.  It  also  stated  that  it  plans  to  develop 
a  systems  engineering  plan  to  address  the  technical  complexities  of  the 
project  in  accordance  with  current  DOD  requirements.  Further,  to  obtain 
an  understanding  of  system  users’  needs,  the  department  stated  that  it 
plans  to  engage  users  and  manage  the  requirements  development  process 
in  accordance  with  our  recommendation.  The  department  stated  that  it 
will  ensure  that  bidirectional  traceability  is  performed  for  all  system 
requirements.  Regarding  its  intent  to  develop  and  document  a  plan  for 
improving  user  satisfaction,  including  identifying  needed  resources  and  a 
schedule  for  improvement,  the  department  stated  that  it  will  augment  its 
current  user  feedback  plan  to  include  these  and  other  key  elements,  such 
as  measurable  outcomes.  Further,  in  response  to  the  need  to  establish 
acquisition  management  processes  in  accordance  with  industry  best 
practices,  at  maturity  level  2,  the  department  said  it  plans  to  establish  a 
milestone  for  completing  the  external  review  in  accordance  with 
Capability  Maturity  Model  guidelines. 

Finally,  the  department  stated  that  it  will  ensure  that  the  six 
recommendations  are  implemented  as  part  of  future  EHR  Way  Ahead 
initiative.  To  the  extent  that  the  department  follows  through  on 
implementing  the  recommendations,  it  should  be  better  positioned  to 
deliver  a  comprehensive  electronic  health  care  record  for  serving  its 
service  members  and  others  entitled  to  military  health  care. 

DOD  also  provided  technical  comments  on  our  draft  report.  In  these 
comments,  DOD  said  it  took  exception  to  several  inaccurate,  misleading, 
and  subjective  statements  provided  in  the  report.  The  department  said  that 
GAO’s  statements  conflicted  with  the  extensive  volume  of  programmatic 
documentation,  written  responses,  and  consistent  interview  feedback 
provided  during  the  course  of  the  audit.  In  particular,  the  department 
believed  that  the  report  did  not  sufficiently  reflect  AHLTA’s  operational 
capabilities  and  its  benefit  to  DOD’s  worldwide  health  care  operations. 
While  we  agree  that  the  department  provided  substantial  documentation, 
we  believe  that  our  analysis  of  the  information  received  supports  our 
findings.  Where  appropriate,  however,  we  have  made  revisions  to 
statements  in  the  report  to  update  our  discussions  of  AHLTA’s  operational 
capabilities  and  the  program’s  management. 
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The  department’s  written  comments  are  reproduced  in  appendix  II.  The 
department  also  provided  technical  comments,  which  we  have 
incorporated  in  the  report  as  appropriate 


We  are  sending  copies  of  the  report  to  appropriate  congressional 
committees,  the  Secretary  of  Defense,  and  other  interested  parties.  In 
addition,  the  report  will  be  available  at  no  charge  on  the  GAO  Web  site  at 
http://www.gao.gov. 

If  you  or  your  staff  have  questions  about  this  report,  please  contact  me  at 
(202)  512-6304  or  melvinv@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Key  contributors  to  this  report  are  listed  in  appendix  III. 

Sincerely  yours, 


Valerie  C.  Melvin 

Director,  Information  Management 


and  Human  Capital  Issues 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


Our  objectives  were  to  (1)  determine  the  Department  of  Defense’s  (DOD) 
status  in  implementing  the  Armed  Forces  Health  Longitudinal  Technology 
Application  (AHLTA)  system,  (2)  determine  the  department’s  plans  for 
acquiring  a  new  electronic  health  record  system,  and  (3)  evaluate  the 
department’s  acquisition  management  for  its  electronic  health  record 
system. 

To  determine  the  department’s  status  in  implementing  the  AHLTA  system, 
we  reviewed  project  status  reports,  acquisition  decision  memorandums, 
quarterly  defense  acquisition  executive  summaries,  monthly  in-progress 
review  reports,  monthly  contractor  performance  reports,  and  overarching 
integrated  project  team  meeting  minutes.  We  supplemented  these  reviews 
with  interviews  of  DOD  officials  in  the  Defense  Health  Information 
Management  System  (DHIMS)  Program  Office,  including  the  DHIMS 
Program  Manager,  Deputy  Program  Manager,  and  Director  of  Products 
Branch  officials  with  whom  we  discussed  the  project’s  cost  and  schedule, 
as  well  as  the  planning,  development,  and  deployment  of  the  original  and 
current  release  of  AHLTA.  We  also  attended  two  demonstrations  of 
AHLTA:  at  the  program  office  located  in  Falls  Church,  Virginia,  and  at  the 
National  Naval  Medical  Center  in  Bethesda,  Maryland.  We  observed 
demonstrations  of  AHLTA  system  functionality  and  held  discussions  with 
system  users.  We  also  observed  a  daily  technical  review  meeting  with 
technical  staff  from  the  Army,  Navy,  and  Air  Force  in  which  the  discussion 
largely  focused  on  the  reporting  of  issues  that  caused  the  system  to  be 
unavailable  to  users  at  various  locations  for  up  to  24  hours.  The  discussion 
also  included  identification  of  known  root  causes  of  the  availability 
problems  (e.g.,  incorrectly  configured  firewalls,  tripped  network  circuits, 
and  problems  with  virtual  private  networks)  and  planned  actions  to 
address  the  issues. 

To  determine  the  department’s  plans  for  acquiring  a  new  system,  we 
reviewed  Electronic  Health  Record  (EHR)  Way  Ahead  planning 
documents.  Specifically,  we  reviewed  the  acquisition  decision 
memorandum  issued  by  the  milestone  decision  authority,  the  Joint 
Requirements  Oversight  Council-approved  Initial  Capabilities  Document 
to  identify  EHR  needs,  and  the  Capabilities-Based  Assessment.  We  also 
reviewed  the  analysis  of  alternatives  procedures  for  guidance  on 
determining  a  technology  solution  for  the  new  EHR.  We  also  reviewed 
department  briefings  issued  between  2008  and  2010,  as  well  as  a  prepared 
statement  to  Congress  from  2009  on  preliminary  plans  for  the  EHR  Way 
Ahead.  These  documents  provided  a  high-level  overview  of  the  need  and 
the  goals  for  the  new  system,  as  well  as  plans  for  the  system’s  enterprise 
architecture  and  expected  capabilities.  We  supplemented  our  review  by 
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interviewing  officials  from  the  EHR  Way  Ahead  planning  office,  including 
the  department’s  Acting  Chief  Information  Officer,  the  DHIMS  Program 
Manager,  and  the  DHIMS  Deputy  Program  Manager. 

To  evaluate  the  department’s  acquisition  management  for  its  electronic 
health  record  system  initiative,  we  evaluated  key  practices  used  by  the 
agency  against  best  practices.  In  this  regard,  we  examined  practices 
related  to  project  management  planning,  systems  engineering  planning, 
system  requirements  development  and  management,  and  user  satisfaction 
improvement  planning  and  compared  the  agency’s  work  with  agency 
policy,  guidance,  and  recognized  best  practices.  Specifically: 

•  To  assess  DOD’s  project  planning  for  AHLTA,  we  compared  the  program’s 
project  management  plan  against  relevant  guidance,  including  the  Military 
Health  System’s  project  management  process  area  description  and  our 
Information  Technology  Investment  Management  framework  for  assessing 
and  improving  process  maturity. 

•  We  assessed  the  agency’s  approach  to  systems  engineering  by  comparing 
program  documentation  such  as  acquisition  strategies  and  the  AHLTA 
project  management  plan  to  systems  engineering  guidance  from  the 
Defense  Acquisition  University  on  systems  engineering.  We  also  reviewed 
relevant  agency  policies,  such  as  DOD  Instruction  5000.02  which  discusses 
the  use  of  systems  engineering  across  the  acquisition  life  cycle  and 
memorandums  from  the  Office  of  the  Under  Secretary  of  Defense  on  a 
2004  revision  to  the  policy  regarding  use  of  a  systems  engineering  plan,  to 
determine  whether  the  AHLTA  program  was  guided  by  appropriate 
systems  engineering  planning  documents  such  as  a  systems  engineering 
plan. 

•  Regarding  requirements  development,  we  reviewed  program  procedures 
describing  the  processes  for  developing  requirements  and  reviewed 
relevant  external  evaluations  of  the  effectiveness  of  those  processes 
against  recognized  guidance.  Specifically,  we  reviewed  an  external 
evaluation  of  the  requirements  development  processes  including  the  2002 
Carnegie  Mellon  External  Assessment  of  the  AHLTA  program  office  and 
the  process  area  description  or  requirements  management.  We  also 
reviewed  the  2008  internal  assessment  of  requirements  management;  a 
2009  concept  of  operations  document  for  a  more  integrated, 
departmentwide  requirements  development  process;  and  the  2010  Joint 
Requirements  Oversight  Council-approved  Initial  Capabilities  Document, 
which  identifies  past  challenges  with  the  department’s  requirements 
processes.  In  addition,  we  analyzed  the  requirements  traceability  matrix 
for  the  most  recent  version  of  AHLTA  to  determine  the  extent  to  which 
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bidirectional  traceability  had  been  performed.  We  also  reviewed  program 
documentation  relative  to  requirements  development  and  user  community 
participation.  In  addition,  we  interviewed  process  improvement  officials, 
including  the  cognizant  official  from  the  Office  of  the  Chief  Information 
Officer  (CIO)  about  internal  acquisition  process  evaluations  and  their 
results  and  the  status  of  plans  for  improving  acquisition  management 
processes.  We  then  compared  the  department’s  current  approach  to 
requirements  development  and  management  with  best  practices  identified 
in  the  Software  Engineering  Institute’s  Capability  Maturity  Model 
Integration  for  Acquisition. 

•  To  assess  the  department’s  approach  to  improving  user  satisfaction,  we 
reviewed  and  analyzed  program  documentation  pertaining  to  the 
collection,  analysis,  and  utilization  of  AHLTA  user  satisfaction  feedback 
such  as  seven  survey  reports  and  a  postimplementation  review  that  were 
produced  between  2005  and  2007  and  compared  the  agency’s  approach  to 
best  practices  such  as  the  Office  of  Management  and  Budget’s  Capital 
Programming  Guide  and  Standards  and  Guidelines  for  Statistical  Surveys. 
We  also  reviewed  lessons  learned  reports  from  2006  through  2008  and  a 
user  conference  briefing  from  2006  that  identified  areas  of  user 
dissatisfaction.  In  addition,  we  reviewed  program  office  documents  that 
identified  improvement  initiatives  such  as  the  AHLTA  3.3  software  release 
and  the  deployment  of  local  cache  servers,  which  were  intended  to 
improve  user  satisfaction.  We  supplemented  our  review  by  interviewing 
program  officials,  including  the  DHIMS  Program  Manager  and  Deputy 
Program  manager,  to  determine  the  extent  to  which  user  satisfaction 
improvement  efforts  and  initiatives  have  been  guided  by  documented 
plans.  We  then  compared  the  department’s  approach  to  improving  user 
satisfaction  with  the  Software  Engineering  Institute’s  IDEALSMl  model, 
which  is  a  recognized  approach  for  managing  process  improvement  efforts 
such  as  managing  improvements  to  user  satisfaction. 

•  To  assess  DOD’s  plans  to  improve  acquisition  management  processes,  we 
reviewed  documentation  and  interviewed  officials  from  the  DHIMS 
Program  Office  and  the  Office  of  Process  Improvement  on  their  plans  to 
improve  the  processes  based  on  the  Software  Engineering  Institute’s 
Capability  Maturity  Model  Integration  for  Acquisition.  We  also  reviewed 


'The  Software  Engineering  Institute  is  a  federally  funded  research  and  development  center 
established  at  Carnegie  Mellon  University  to  address  software  engineering  practices. 

IDEAL  is  a  service  mark  of  Carnegie  Mellon  University  and  stands  for  initiating,  diagnosing, 
establishing,  acting,  and  leveraging.  For  more  information  on  this  model,  see  IDEALSM:  A 
User’s  Guide  for  Software  Process  Improvement  (CMU/SEI-96-HB-001). 
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DOD’s  2008  internal  assessment  related  to  acquisition  management 
processes,  action  plans,  and  tasks  planned  for  process  improvement. 

We  supplemented  our  analysis  with  interviews  with  officials  in  the  DHIMS 
Office,  including,  the  Program  Manager,  Deputy  Program  Manager, 
Director  of  Products  Branch  and  Engineering  and  Resources  offices.  We 
also  obtained  written  responses  from  the  responsible  program  manager  or 
subject  matter  expert  for  areas  of  our  review.  These  responses  were 
approved  by  the  MHS  CIO  or  the  Program  Executive  Officer,  Joint  Medical 
Information  Systems/Deputy  MHS  CIO. 

We  did  not  conduct  an  independent  validation  of  the  life-cycle  costs  and 
obligations  provided  to  us  by  DOD. 

We  conducted  this  performance  audit  at  the  DHIMS  Program  Office  in 
Falls  Church,  Virginia,  and  the  National  Naval  Medical  Center,  in 
Bethesda,  Maryland,  from  September  2009  through  October  2010  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  objectives. 
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OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
HEALTH  AFFAIRS 

SKYLINE  FIVE,  SUITE  810,  8111  LEESBURG  PIKE 
FALLS  CHURCH,  VIRGINIA  22041-3206 


I  RK  AM, 

MANAGEMENT 

ACTIVITY 

Ms.  Valerie  C.  Melvin 

Director,  Information  Management 

and  Human  Capital  Issues 

U.S.  Government  Accountability  Office 

441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Ms.  Melvin: 


2  7  2010 


This  is  the  Department  of  Defense  (DoD)  response  to  the  recommendations  in 
the  Government  Accountability  Office  (GAO)  Draft  Report  GAO- 11-50,  “Information 
Technology  -  Opportunities  Exist  to  Improve  Management  of  DoD’s  Electronic  Health 
Record  Initiative,”  October  2010  (Engagement  Code  310944). 

DoD  acknowledges  receipt  of  the  draft  report  and  will  address  each  of  the 
recommendations  and  ensure  appropriate  measures  are  carried  out  effectively.  DoD  takes 
exception  to  several  inaccurate,  misleading,  and  subjective  statements  provided  in  the 
draft  report.  GAO’s  statements  conflict  with  the  extensive  volume  of  programmatic 
documentation,  written  responses,  and  consistent  interview  feedback  provided  during  the 
course  of  this  audit.  Enclosed  are  suggested  technical  comments  and  corrections  to 
GAO’s  draft  report. 

Thank  you  for  the  opportunity  to  review  and  comment  on  the  draft  report.  The 
points  of  contact  for  additional  information  are  Ms.  Lois  Kellett, 
Lois.Kellett@tma.osd.mil,  or  (703)  681-8836;  and  Mr,  Gunther  Zimmerman, 
Gunther.Zimmerman@tma.osd.mil,  or  (703)  681-4360. 


/?. _ y 


George  Peach  Taylor,  Jr.,  M.D. 

Deputy  Assistant  Secretary  of  Defense 
(Force  Health  Protection  and  Readiness) 
Performing  the  Duties  of  the 
Assistant  Secretary  of  Defense 
(Health  Affairs) 


Attachments: 
As  stated 
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GAO  Draft  Report-Dated  OCTOBER  2010 
GAO-11-50  (Engagement  Code  310944) 

“Opportunities  Exist  to  Improve  Management  of  DoD’s  Electronic  Health  Record 

Initiative” 

Department  of  Defense  Comments  to  GAO  Recommendations 

RECOMMENDATION :  Develop  and  maintain  a  comprehensive  project  plan  that 
includes  key  elements,  such  as  the  project’s  scope,  cost,  schedule  and  risks  and  update 
the  plan  to  provide  key  information  for  stakeholders  on  the  project’s  plans  and  status. 

DoD  RESPONSE:  Concur.  DoD  will  develop  and  maintain  a  comprehensive  project 
plan  in  accordance  with  this  recommendation  and  DoD  acquisition  program  guidelines. 

RECOMMENDATION:  Develop  a  systems  engineering  plan  in  accordance  with  DoD 
guidance  to  address  the  technical  complexities  of  delivering  a  worldwide  electronic 
health  record  system. 

DoD  RESPONSE:  Concur.  Since  implementing  a  requirement  to  develop  a  Systems 
Engineering  Plan  (SEP)  in  February  2004,  DoD  will  continue  to  develop  and  maintain  a 
SEP  in  accordance  with  this  recommendation. 

RECOMMENDATION:  Ensure  that  the  requirements  development  process  involves 
system  users  throughout  the  development  process,  to  obtain  an  understanding  of  what 
will  satisfy  their  needs. 

DoD  RESPONSE:  Concur.  DoD  will  continue  to  engage  system,  users  and  manage  the 
requirements  development  process  in  accordance  with  this  recommendation. 

RECOMMENDATION :  Ensure  the  establishment  of  bidirectional  traceability  for  all 
system  requirements. 

DoD  RESPONSE:  Concur.  DoD  will  ensure  the  bidirectional  traceability  for 
requirements  in  accordance  with  this  recommendation. 
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RECOMMENDATION :  Develop  and  document  a  plan  for  improving  user  satisfaction 
that  prioritizes  improvement  projects;  identifies  needed  resources;  includes  schedules  for 
improvement  efforts,  including  future  user  feedback  survey;  and  links  efforts  to 
measureable  outcomes  and  specific  user  needs. 

DoD  RESPONSE:  Concur.  DoD  will  augment  its  current  user  feedback  plan  for 
improved  user  satisfaction  in  accordance  with  this  recommendation. 

RECOMMENDATION :  Establish  acquisition  management  processes  in  accordance 
with  industry  best  practices,  including  indentifying  milestones  and  a  completion  date  for 
the  external  evaluation  that  MHS’s  processes  are  at  maturity  Level  2  of  the  Capability 
Management  Maturity  Model  Integrated  for  Acquisition. 

DoD  RESPONSE:  Concur.  DoD  continues  to  follow  DoD  acquisition  program 
guidelines  and  will  establish  a  milestone  for  an  external  review  in  accordance  with  the 
Capability  Management  Maturity  Model  guidelines. 

RECOMMENDATION:  Further,  to  help  ensure  that  the  EHR  Way  Ahead  does  not 
have  shortfalls  similar  to  those  experienced  with  AHLTA,  we  recommend  that  the  above 
management  practices  be  implemented  as  part  of  the  planning  for  this  important 
initiative. 

DoD  RESPONSE:  Concur.  DoD  will  ensure  that  these  recommendations  are 
implemented  as  part  of  the  EHR  Way  Ahead  initiative. 
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Valerie  C.  Melvin,  (202)  512-6304  or  melvinv@gao.gov 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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